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DETAILED ACTION 

A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1. 17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on 4/1 1/2005 has been entered. Claims 1,6, 1 1, 26- 
29 have been amended. Claims 3-4, 9-10,14-25 have been canceled. Claims 1-2, 5-8, 11-13 and 
26-29 are pending in the present application. 



Response to Arguments 

Applicant's arguments filed Sept. 27, 2004 have been fully considered but they are not 
persuasive. As addressed below, Xiong, Shum and Teo teach the claim limitations. 

Applicant argues in essence with respect to the claim 1 and similar claims that Teo does 
not teach automatically setting the mapping mode. However, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that automatically provides feedback to the computer 
system when the pair-wise objective function is not desirable for a poor selection of the 
projection viewing plane and the resulting panoramas will have imperfectly aligned images that 
give shadow or ghosting effects due to a poor selection of the projection viewing plane and the 
resulting panoramas will have imperfectly aligned images that give shadow or ghosting effects. 
Xiong further teaches a human interaction being present throughout the image synthesis process 
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to provide feedback to the computer system in all the nonlinear optimizations to let users monitor 
the progress of the system and allow them to intervene when necessary . 

Xiong discloses in column 8, lines 18-58 that a panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen 
viemne plane (a geometric surface) such as cubic, polyhedral, cylindrical and spherical 
geometries. Xiong describes that a user can alternatively intervene to select a viewing plane. 
Therefore Xiong at least suggests automatically setting a mapping surface without a user 
intervention to determine the mapping mode. 

Teo teaches that the viewing software can automatically invert the projection by 
automatically straightening out the portion of the panoramic image being displayed and thereby 
automatically setting the mapping mode by the viewer software (column 7-8). Furthermore, the 
claim limitation of "automatically setting the mapping mode" does not necessarily imply 
automatically setting the mapping surface. Teo teaches the image processor for automatically 
transforming the first digital panoramic image into a second digital panoramic image 
corresponding to the projection of the scene onto the second surface geometry and therefore 
automatically setting the mapping mode of mapping the images to the selected surface geometry 
(column 7). Teo teaches in column 10 the mapping mode of mapping onto the planar surfaces 
such as mapping the panorama images onto a cube and the mapping mode of mapping the 
panorama images onto polyhedral surfaces to form cylindrical panorama images. Once the 
geometry is selected either by a user or by the viewer software, the image processor 
automatically transforms the first digital panoramic image into a second digital panoramic image 
corresponding to the projection of the scene onto the surface geometry. Therefore, the mapping 
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mode is automatically set by the viewer software because the steps involved in the mapping or 
transforming the first digital panoramic image into a second digital panoramic image 
corresponding to the projection of the scene onto the changed/selected geometric surface are 
automatically performed by the software. Although Teo also teaches a user can intervene in 
selecting a mapping surface, however, the mapping mode of mapping the first digital panoramic 
image into a second digital panoramic image corresponding to the projection of the scene onto a 
selected geometric surface is automatically set by the software. It is also noted that each mapping 
mode of transforming the panoramic scene corresponds to a selected geometric surface. 

Although Teo teaches that a user can change a mapping surface ( column 11, lines 1-9 ) by 
adjusting the shape of the mapping surfaces so that a given cylindrical panoramic image can be 
transformed to produce a modified panoramic image, the procedure can be automated by the 
computer software. Teo therefore teaches automatically setting a mapping mode or changing the 
plane for mapping the panoramic image by the computer through a user interface or by the 
computer software without the user interface. 

Teo further discloses the horizontal placement of image by projecting a scene onto a cube 
so that horizontal lines of the scene appear horizontal in the panoramic image, and vertical 
placement of image by projecting the panoramic image on the vertical cylinder or by rotating the 
polyhedral surface relative to a cylindrical panoramic image and making adjustments to the sides 
and edges of the polyhedral surface so that the placement of images appear vertically on the 
mapping surface. Teo teaches the user interface enabling the user to rotate the polyhedral surface 
relative to a cylindrical panoramic image and making adjustments to the sides and edges of the 
polyhedral surface so that the placement of images appear vertically on the mapping surface. 
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Shum teaches a plurality of mapping modes such as zooming and panning the screen 
images of the interior scene shown in the panoramic view of Fig. 10A wherein the mapping 
surfaces within panoramas are different (See Figs. 1 1A and 1 IB and column 23-24) and 
changing the type of panoramas such as the spherical panoramas (Figs. 15A-15B) and the planar 
panoramas (Figs. 16A-16B). Shum teaches a setting step of zooming/panning the panorama and 
setting the spherical panoramas (column 23-24) by the interactive user interface so that the 
mapping mode changes in which each resulting panorama has different mapping surfaces 
(column 23-24). Shum further teaches determining the constraints governing the mapping mode 
of the mapping onto the projection plane wherein the constraints incorporates the predetermined 
panoramic view of a 3D scene or the predetermined camera orientation matrix by the governing 
linear systems of equations. Shum further teaches generating a warning message to indicate that 
the constraint equations are unsolvable in a case in which the synthesized image is not aligning 
with any portion of the projected model in response to the change in the geometric constraints 
determined by the projection plane or in response to the change in the panoramic view. The 
geometric constraints change due to the change in the projection plane from a planar panorama 
to a spherical panorama wherein the planes within the spherical panorama are changed to 
spherical planes and the panorama view can be changed as well by zooming/panning panorama 
and/or selecting a portion of the panorama. Therefore, un-solvability of the linear equations 
occurs in the process of changing the mapping mode because the change in the mapping mode or 
the change in the geometric surface also changes the geometric constraints governing the linear 
equations. When the linear systems of equations are not solvable, a user can switch the 
mapping plane orientation or changing the mode of projection surface to meet the 
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constraint equations requirements wherein the plane normals have changed. Moreover, the 
projection space can be in the form of any shape such as spherical or planar and by changing the 
camera's point of view and/or the panorama view and the coordinates and directions of the plane 
normals, the projection space is effectively changed and accordingly the mapping mode is 
changed ( column 13-24) . 

However, Shum teaches changing the mapping mode by zooming and panning the screen 
images of the interior scene shown in the initial panoramic view of Fig. 10A (See Figs. 1 1A and 
1 IB and column 23-24) and changing the type of panoramas such as the spherical panoramas 
(Figs. 15A-15B) and the planar panoramas (Figs. 16A-16B). Shum teaches a setting step of 
zooming/panning the panorama and setting the spherical panoramas (column 23-24) by the 
interactive user interface so that the mapping mode changes wherein each resulting panorama 
has different mapping surfaces in accordance with the user input commands (column 23-24) 
wherein the panorama mapping surfaces change as a result of the zooming and panning. It is 
noted that this user interface can be automated by the computer software. 

Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-2, 5-8, 11-13, and 26-29 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Xiong et al. U.S. Patent No. 6,434,265 (hereinafter Xiong), and further in view 
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of Shum et al. U.S. Patent No. 6,271,855 (hereinafter Shum) and Teo U.S. Patent No. 6,246,413 
(Teo). 

3. Reclaims 1 and 26: 

(a) Xiong teaches an image synthesis method comprising: 

An input step, of inputting a plurality of image data representing a plurality of images 
(e.g., column 9, lines 20-30); 

A placement information generating step, of generating placement information about the 
horizontal and vertical placement direction of the plurality of images, determined by the 
horizontal and vertical placement direction of the plurality of images represented by the image 
data inputted in the input step (e.g., column 16, lines 35-45); 

A placement information obtaining step of obtaining the placement information about a 
plurality of images in which adjacent images have a common subject region (e.g., in column 4, 
lines 5-40, Xiong teaches a method for constructing a panorama from rectilinear images in 3D 
through projective registration and calibration including: (1) the projective registrations of 
overlapping images, (2) calibration and global optimization of these images, a self calibration in 
which 2D image planes are positioned as 3D planes in space); 

A setting step of setting one mapping mode out of a plurality of mapping modes, each 
mapping mode corresponding to a different mapping surface (e.g., in column 8, lines 18-58 of 
Xiong, it is stated that (< overlapping photographs are analyzed to determine what orientation the 
photographs were taken in order to establish a common ground for subsequent operations and 
the panorama is constructed on a particular geometry that will best facilitate the rendering of 
the projection of the panorama onto a chosen viewing plane for viewing". The Xiong discloses 
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some typical geometry on which panoramas are formed; In column 8, lines 18-58, Xiong further 
discloses that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane such as cubic, polyhedral cylindrical 
and spherical geometries)', and 

A synthesis step of combining the plurality of images by using the mapping mode set in 
the setting step (e.g, in column 4, lines 5-40 of Xiong, it is stated that the composing or blending 
in which images are ready to be re-projected to a 3D environment map with pixels in overlap 
regions being composed from multiple; In column 8, lines 18-58, Xiong further teaches that 
overlapping photographs are analyzed to determine what orientation the photographs were 
taken in order to establish a common ground for subsequent operations and the panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection of 
the panorama onto a chosen viewing plane for viewing. The Xiong further discloses some typical 
geometry on which panoramas are formed)', 

A changing step, of changing the mapping mode (In column 4, lines 40-50, it is stated 
that "the projection module may be controlled through the user interface 230 as well, to allow a 
user to select what geometry will be projected onto Therefore, Xiong teaches a changing step 
through the user interface 230 by selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface). 

- The Examiner interprets "a placement information obtaining step of obtaining placement 
information about a plurality of images in which adjacent images have a common subject 
region" as an automatic registration and calibration step of registering the overlapping 
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images and capturing common overlapping areas between overlapping images and 
minimizing the average squared pixel intensity difference with respect to certain 
transformation parameters. 
- The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
mapping modes each corresponding to a different mapping surface in accordance with 
said obtained placement information. 

(b) However, Xiong is silent to "a setting step, of automatically setting one mapping 
mode out of a plurality of mapping modes, each mapping mode corresponding to a different 
mapping surface, without a user intervening to determine the mapping mode or the corresponding 
mapping surface, in accordance with the horizontal and vertical placement direction of the 
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plurality of images having a common subject region obtained in the placement information 
obtaining." 

(c) Teo teaches the claim limitation of a setting step, of automatically setting one 
mapping mode out of a plurality of mapping modes, each mapping mode corresponding to a 
different mapping surface, without a user intervening to determine the mapping mode or the 
corresponding mapping surface, in accordance with the horizontal and vertical placement 
direction of the plurality of images having a common subject region obtained in the placement 
information obtaining (Teo teaches that the viewing software can automatically invert the 
projection by automatically straightening out the portion of the panoramic image being 
displayed and thereby automatically setting the mapping mode by the viewer software in column 
7-8. Furthermore, Teo teaches the image processor for automatically transforming the first 
digital panoramic image into a second digital panoramic image corresponding to the projection 
of the scene onto the second surface geometry and therefore automatically setting the mapping 
mode of mapping the images to the selected surface geometry in column 7. Teo teaches in 
column 10 the mapping mode of mapping onto the planar surfaces such as mapping the 
panorama images onto a cube and the mapping mode of mapping the panorama images onto 
polyhedral surfaces to form cylindrical panorama images. Teo teaches horizontal placement of 
image by projecting a scene onto a cube so that horizontal lines of the scene appear horizontal 
in the panoramic image and vertical placement of image by projecting the panoramic image on 
the vertical cylinder or by rotating the polyhedral surface relative to a cylindrical panoramic 
image and making adjustments to the sides and edges of the polyhedral surface so that the 
placement of images appear vertically on the mapping surface. Although Teo discloses user may 
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intervene in changing a mapping surface, e.g., Teo teaches that a user can change a mapping 
surface, see column 11. lines 1-9. Teo does not teach away from the claim limitation of "without 
a user intervening to determine the mapping mode " because a user changes a mapping surface, 
not a mapping mode. The mapping mode is automatically determined by the viewer software. 
Moreover, any manual procedure of changing a mapping surface can be converted to a 
software-automated procedure. See In re Venner, 120 USPQ 192, 194; 262 F2d 91 (CCPA 
1958)). 

(d) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Teo's setting step because Xiong discloses in column 8, lines 18-58 that a panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection 
onto a chosen viewins plane (a geometric surface) such as cubic, polyhedral, cylindrical and 
spherical geometries. Xiong describes that a user can alternatively intervene to select a viewing 
plane. Therefore Xiong at least suggests automatically setting a mapping surface without a user 
intervention to determine the mapping mode. 

(e) One having the ordinary skill in the art would have been motivated to do this because 
setting a mapping mode out from a plurality of mapping modes serves for the purpose of 
eliminating visible artifacts and providing a best projection (Teo column 10). 

(f) However, Xiong and Teo are silent to issuing warning wherein the warning being 
issued in a case in which the synthesized image exceeds a predetermined angle of view when a 
cylindrical mapping mode is changed to a plane mapping mode in the claim limitation of "a 
generating step, of issuing , when an image formed by changing the mapping mode in the changing 
step does not comply with a predetermined condition set in accordance with the mapping mode, a 
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warning and generating a synthesized image in accordance with the predetermined condition, the 

warning being issued in a case in which the synthesized image exceeds a predetermined angle of 

view" (i.e., generation of a warning message). 

(g) Shum implicitly teaches issuing warning wherein the warning being issued in a case 
in which the synthesized image exceeds a predetermined angle of view when a cylindrical 
mapping mode is changed to a plane mapping mode in the claim limitation of "a generating step, 
of issuing , when an image formed by changing the mapping mode in the changing step does not 
comply with a predetermined condition set in accordance with the mapping mode, a warning and 
generating a synthesized image in accordance with the predetermined condition, the warning 
being issued in a case in which the synthesized image exceeds a predetermined angle of view 
while simultaneously generating and displaying the synthesized image within the predetermined 
range of viewing angle" (e.g., Shum teaches a plurality of mapping modes such as zooming and 
panning screen images of the interior scene shown in the panoramic view of Fig. 10A. See Figs. 
11 A and 11B and column 23-24. Shum also teaches the type of panoramas such as the spherical 
panoramas in Figs. 15A-15B and the planar panoramas in Figs. 16A-16B. Shum teaches a 
setting step of zooming/panning the panorama and setting the spherical panoramas in column 
23-24 by the interactive user interface so that the mapping mode changes wherein each resulting 
panorama has different mapping surfaces in accordance with the user input commands. Shum 

further teaches determining the constraints governing the mapping mode of the mapping onto the 
projection plane wherein the constraints incorporates the predetermined angle of view such as 
the panoramic view of a 3D scene or the predetermined camera orientation matrix by the 
governing linear systems of equations. Shum further teaches generating a warning message to 
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indicate that the constraint equations are unsolvable in a case in which the synthesized image is 
not aligning with any portion of the projected model when the projection plane is changed or 
when the angle of view is changed in response to the user commands, e.g., the projection plane is 
changed from a planar panorama to a spherical panorama wherein the planes within the 
spherical panorama are changed to spherical planes and the panorama view can be changed as 
well by zooming/panning panorama and/or selecting a portion of the panorama. When the linear 
systems of equations are not solvable, a user can switch the mapping plane orientation or 
changing the mode of projection surface to meet the constraint equations requirements by 
changing the plane normals. Moreover, the projection space can be in the form of any shape 
such as spherical or planar and by changing the camera 's point of view and/or the panorama 
view and the coordinates and directions of the plane normals, the projection space is effectively 
changed and thereby the mapping mode is changed. See column 13-24. ) 

(h) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Shum's warning message generation step because Xiong suggests a generating step of generating 
a synthesized image in accordance with the predetermined condition (Xiong column 3, lines 35- 
55; column 17, lines 15-67; column 18, lines 1-4). Moreover, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that provides feedback to the computer system such as 
issuing warning messages on the computer monitor 218 when the pair-wise objective function is 
not desirable for a poor selection of the projection viewing plane (and the resulting panoramas 
will have imperfectly aligned images that give shadow or ghosting effects) due to a poor 
selection of the projection viewing plane and the resulting panoramas will have imperfectly 
aligned images that give shadow or ghosting effects. Xiong also points to a seamless multi- 
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resolution average blending method that would result in an absent of shadow effects (column 14, 
lines 1-45). Xiong further teaches a human interaction being present throughout the image 
synthesis process to provide feedback to the computer system in all the nonlinear optimizations 
to let users monitor the progress of the system and allow them to intervene when necessary and 
therefore the claimed limitation suggests an obvious modification of Xiong because providing 
feedback to users is similar to issuing a message to users in a user interaction with the nonlinear 
optimizations when the changing step does not comply with a predetermined condition set or 
user-selected parameter set. 

(i) One having the ordinary skill in the art would have been motivated to do this because 
this would have provided the user the informative message such as issuing a warning message 
through the user interaction when convergence to the predetermined condition setting cannot be 
obtained in the image synthesis (Xiong column 17, lines 15-67, column 18, lines 1-4) for non- 
solvable optimization problems (Shum column 18, lines 38-67). 

Claim 2: 

Claim 2 recites all the limitations of claim 1 and adds the limitation of "a focal length 
obtaining step." Xiong teaches finding projective parameters such as 3D rotation parameters 
(pan, tilt roll), center of projection of images, ratio of focal lengths, and the like (column 10, 
lines 9-28). Xiong implicitly teaches finding the camera internal parameters vector of which the 
focal length is a component (column 1 1 , lines 1 5-42). 



Claim 5: 
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The claim 5 recites all the limitations of claim 1 and adds the limitation of "a displaying 
step of displaying a cuttable rectangular region " Xiong teaches how to align images more 
precisely by changing the coordinates for positioning an image. Xiong further teaches placing the 
images 1210 at selected tangents on the viewing sphere 1220 (figure 12, and column 17, lines 
12-65). 

4. Claim 6: 

The claim 6 encompasses the same scope of invention as that of claim 1 except additional 
claimed limitation of "an image synthesis apparatus" and "a displaying step of displaying a 
cuttable rectangular region". However, Xiong further discloses in figure 3 an apparatus for image 
synthesis. Xiong teaches how to align images more precisely by changing the coordinates for 
positioning an image. Xiong further teaches placing the images 1210 at selected tangents on the 
viewing sphere 1220 (figure 12, and column 17, lines 12-65). 

Claim 7: 

The claim 7 recites all the limitations of claim 6 and adds the limitation of "a focal 
length obtaining step." The Xiong reference teaches finding projective parameters such as 3D 
rotation parameters (pan, tilt roll), center of projection of images, ratio of focal lengths, and the 
like (column 10, lines 9-28). Xiong implicitly teaches finding the camera internal parameters 
vector in his image synthesis apparatus of figure 3 because the focal length is a component of 
that vector (column 11, lines 15-42). 

Claim 8: 
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The claim 8 recites all the limitations of claim 6 and adds the limitation of "a changing 
step of changing the mapping mode." Xiong teaches that panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane such as cubic, polyhedral, cylindrical and spherical geometries (column 8, lines 
18-58). 



5. Claim 11: 

The claim 1 1 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method. However, Xiong further discloses the claimed limitation 
of a computer-readable storage medium having a program for implementing image synthesis 
method (e.g., in column 3, lines 54-57, it is stated "a program residing in system memory 220 
which stores output data and other data"). 

Claim 12: 

The claim 12 recites all the limitations of claim 1 1 and adds "a focal length obtaining 
step." Xiong teaches finding projective parameters such as 3D rotation parameters (pan, tilt roll), 
center of projection of images, ratio of focal lengths, and the like (column 10, lines 9-28). Xiong 
implicitly teaches finding the camera internal parameters vector of which the focal length is a 
component (column 11, lines 15-42). 

Claim 13: 

The claim 13 recites all the limitations of claim 1 1 and adds the limitation of "a changing 
step of changing the mapping mode " Xiong teaches that panorama is constructed on a particular 
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geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane such as one of the cubic, polyhedral, cylindrical and spherical geometries (column 
8, lines 18-58). 

6. Claim 27: 

The claim 27 encompasses the same scope of invention as that of the claim 26 except 
additional claim limitation of "a displaying step of displaying a cuttable rectangular region." 
Xiong teaches how to align images more precisely by changing the coordinates for positioning 
an image. Xiong further teaches placing the images 1210 at selected tangents on the viewing 
sphere 1220 (figure 12, and column 17, lines 12-65). 

7. Claim 28: 

The claim 28 encompasses the same scope of invention as that of claim 26 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method. However, Xiong further discloses the claimed limitation 
of a computer-readable storage medium having a program for implementing image synthesis 
method (e.g., in column 3, lines 54-57, it is stated "a program residing in system memory 220 
which stores output data and other data"). 

Claim 29: 

The claim 29 encompasses the same scope of invention as that of the claim 1 . The claim 
29 is subject to the same rationale of rejection set forth in the claim 1 . 
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Conclusion 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (571) 272-7664. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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